Guest lecture: Experiments

Damon C. Roberts, University of Colorado

Who am I?

PhD Candidate at CU

Specializes in:
- Political psychology
  - Often in the context of the United States
- Political neuroscience
- Applied Statistics
  - Experiments
  - Surveys
  - Bayesian Statistics
  - Missing data problems

Motivations for using experiments

Causal inference

Often in science we engage in causal inference.
- We do not want to just examine whether there is a correlation between two variables, but we often want to to examine the causal relationship.

Correlational relationships

Two variables may be related, but the relationship may just be spurious.

Table 1: Cross tabulation
WaffleHouses_Cat		Much lower than average	Lower than average	Higher than average	Much Higher than average	All
No	N	7	9	7	2	25
	% row	28.0	36.0	28.0	8.0	100.0
Yes	N	1	8	11	5	25
	% row	4.0	32.0	44.0	20.0	100.0
All	N	8	17	18	7	50
	% row	16.0	34.0	36.0	14.0	100.0
Data source: Waffles dataset (McElreath 2020). Credit: damoncroberts.com

Nicholas Cage films

Arcades

Causal relationships

Variables causing some outcome

%%{init: {'theme':'dark', 'themeVariables':{'primaryColor':'#000000', 'primaryBorderColor': '#fffffff'}}}%%
graph LR
A["☀️"] --> B["🌡️"]

Figure 1: Simple causal relationship

However, detecting causal relationships is quite complicated!

%%{init: {'theme':'dark', 'themeVariables':{'primaryColor':'#000000', 'primaryBorderColor': '#fffffff'}}}%%
graph LR
A["☀️"] --> B["🌡️"]
C["☁️"] --> A
C --> B

Figure 2: Confounded hypothesis

In science we are often interested in isolating the effect of a primary independent variable on a dependent variable.
- There are often two primary strategies we can try to do this with.

Internal versus external validity

Internal validity

Changes in your dependent variable are not caused by anything but your independent variable.
There is not omitted variable bias (a confounding variable you have not controlled for).
Often comes with experimental designs
- Only one thing changes in the environment, so any change that occurs in your dependent variable is likely because of what you changed in the environment.

External validity

Changes in your dependent variable can be caused by multiple factors.
Beneficial in that this is more realistic.
- It is rarely the case that only one thing influences your dependent variable and only one thing changes at a time.
Often comes with observational designs
- You just observe what is happening in the world and collect data on it. Changes that occur in your outcome are not “artificial” and are happening in the real world.

Often times there is some trade-off between these.
- If we choose a design that is highly internally valid, we have less external validity. And vice versa.

Experiments

How do they work?

We need to define the main independent variable of interest that we think explains our dependent variable.

We can control the environment someone is in and will only allow for the independent variable to change.
- We often randomly assign people into different values of the independent variable
In the context of an experiment, this independent variable that we manipulate is called a treatment.
In many cases: treatment \(\neq\) independent variable
- For example: if party identification is our treatment, we often cannot change someone’s partisanship in a lab very easily! We can get close by changing motivations to rely on their partisanship, though.
- This is often one way that experiments are low on external validity.
It increases internal validity by: making sure that the only thing changing is the treatment and then looking at how the dependent variable changes as a result. It ensures that we don’t all of the real world noise going on around this.

A simple example of how an experiment works

Participants come into a lab
They fill out some paperwork that ensures they know their rights and lets them know what to expect from the experience
Half of the participants are randomly assigned to take a sugar pill (a placebo), the second is randomly chosen to take the anti-depressant
Participants do not know what “treatment” they are being given. Neither do the researchers. Which “treatment condition” someone is assigned to is entirely random.
The researchers then document how everyone’s rates of depression changes after taking one of the two pills.

A political science example of how an experiment works

From Roberts and Wolak (2022).
A researcher wants to examine whether people are more likely to vote for younger candidates relative to older political candidates.

Treatment:
- Placebo condition: the candidate is described as 50 years old (roughly the average age of a member of office).
- Young condition: the candidate is described as 25 years old.
- Old condition: the candidate is described as 75 years old.

Placebo condition

Jack Milhone is running for the state legislature in 2020. Milhone is a 50-year-old resident of Jefferson County. A graduate of Hanover College, Milhone currently works at an insurance brokerage firm. Milhone describes his priorities as reforming school funding and promoting job growth in the state. Milhone says he is running for the state legislature because, “I want the people in my community to know that their voice will be heard in state government.” The average age of lawmakers in the state legislature is 52.¹

Young condition

Jack Milhone is running for the state legislature in 2020. Milhone is a 23-year-old resident of Jefferson County. A graduate of Hanover College, Milhone currently works at an insurance brokerage firm. Milhone describes his priorities as reforming school funding and promoting job growth in the state.Milhone says he is running for the state legislature because, “I want the people in my community to know that their voice will be heard in state government.” If elected, Milhone will be the youngest lawmaker in the state legislature.¹

Old condition

Jack Milhone is running for the state legislature in 2020. Milhone is a 77-year-old resident of Jefferson County. A graduate of Hanover College, Milhone currently works at an insurance brokerage firm. Milhone describes his priorities as reforming school funding and promoting job growth in the state. Milhone says he is running for the state legislature because, “I want the people in my community to know that their voice will be heard in state government.” If elected, Milhone will be the oldest lawmaker in the state legislature.¹

People are randomly assigned to see one of these three things.
They only see one of them.

They are then asked an operationalized measure of willingness to vote for the candidate:

Imagine that you live in the state where Jack Milhone is running for state legislator. How likely is it that you would vote for him in the upcoming election? 0 Very unlikely, 100 Very likely

Dissertation example

Experiments allow you to get really creative!
They aren’t just used using what are called “vignettes” like the example above!

In this study, I am trying to test the following hypotheses:
1. People notice color
2. Colors shape the perceptions of candidates based on partisan perceptions.
3. Candidates using Red are more supported by Republicans; candidates using Blue are more supported by Democrats.
4. Republicans spend less time coming to evaluations of candidates using Red; Democrats spend less time evaluating candidates using Blue.

References

Roberts, Damon C., and Jennifer Wolak. 2022. “Do Voters Care about the Age of Their Elected Representatives?” Political Behavior.